Red wine quality project: Shapley values decomposition

Jakub Kosterna, Bartosz Siński, Jan Smoleń

Packages load & generation seed declaration

import pandas as pd
import numpy as np

import pickle

import dalex as dx

np.random.seed = 42

X_test = pd.read_csv("../data/X_test_chosen_ones.csv").drop(['Unnamed: 0', 'test set id'], axis=1)
y_test = pd.read_csv("../data/y_test_chosen_ones.csv").drop(['Unnamed: 0', 'test set id'], axis=1)
observations_info = pd.read_csv("../data/chosen_obserwations_basic.csv").drop('Unnamed: 0', axis=1)

gbm = pickle.load(open("../models/gbm.pickle", 'rb'))
svm = pickle.load(open("../models/svm.pickle", 'rb'))
rfc = pickle.load(open("../models/rfc.pickle", 'rb'))
gbc = pickle.load(open("../models/gbc.pickle", 'rb'))

explainer_gbm = dx.Explainer(gbm, X_test, y_test, label = "XGBoost")
explainer_svm = dx.Explainer(svm, X_test, y_test, label = "Support Vector Machine")
explainer_rfc = dx.Explainer(rfc, X_test, y_test, label = "Random Forest")
explainer_gbc = dx.Explainer(gbc, X_test, y_test, label = "Gradient Boosting")

Preparation of a new explainer is initiated

  -> data              : 16 rows 11 cols
  -> target variable   : Parameter 'y' was a pandas.DataFrame. Converted to a numpy.ndarray.
  -> target variable   : 16 values
  -> model_class       : sklearn.model_selection._search.RandomizedSearchCV (default)
  -> label             : XGBoost
  -> predict function  : <function yhat_proba_default at 0x000001C20CDBF700> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 0.151, mean = 0.565, max = 0.923
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -0.894, mean = -0.0651, max = 0.849
  -> model_info        : package sklearn

A new explainer has been created!
Preparation of a new explainer is initiated

  -> data              : 16 rows 11 cols
  -> target variable   : Parameter 'y' was a pandas.DataFrame. Converted to a numpy.ndarray.
  -> target variable   : 16 values
  -> model_class       : sklearn.model_selection._search.GridSearchCV (default)
  -> label             : Support Vector Machine
  -> predict function  : <function yhat_default at 0x000001C20CDBF670> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 0.0, mean = 0.5, max = 1.0
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -1.0, mean = 0.0, max = 1.0
  -> model_info        : package sklearn

A new explainer has been created!
Preparation of a new explainer is initiated

  -> data              : 16 rows 11 cols
  -> target variable   : Parameter 'y' was a pandas.DataFrame. Converted to a numpy.ndarray.
  -> target variable   : 16 values
  -> model_class       : sklearn.model_selection._search.RandomizedSearchCV (default)
  -> label             : Random Forest
  -> predict function  : <function yhat_proba_default at 0x000001C20CDBF700> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 0.246, mean = 0.551, max = 0.898
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -0.848, mean = -0.0505, max = 0.754
  -> model_info        : package sklearn

A new explainer has been created!
Preparation of a new explainer is initiated

  -> data              : 16 rows 11 cols
  -> target variable   : Parameter 'y' was a pandas.DataFrame. Converted to a numpy.ndarray.
  -> target variable   : 16 values
  -> model_class       : sklearn.model_selection._search.GridSearchCV (default)
  -> label             : Gradient Boosting
  -> predict function  : <function yhat_proba_default at 0x000001C20CDBF700> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 0.0168, mean = 0.51, max = 0.962
  -> model type        : classification will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -0.93, mean = -0.00993, max = 0.983
  -> model_info        : package sklearn

A new explainer has been created!

for row in range(16):
    print(observations_info.iloc[row]["details"])
    pp_bd_gbm = explainer_gbm.predict_parts(X_test.iloc[row,:], type = "shap")
    pp_bd_gbm.plot()

all TP #1

all TP #2

all TP #3

all TN #1

all TN #2

all TN #3

all FN #1

all FN #2

all FP #1

all FP #2

XGBoost TP, others FN

XGBoost TN, others FP

SVM TP, others FN

SVM TN, others FP

Random Forest TN, others FP

Gradient Boosting TP, others FN

for row in range(16):
    print(observations_info.iloc[row]["details"])
    pp_bd_svm = explainer_svm.predict_parts(X_test.iloc[row,:], type = "shap")
    pp_bd_svm.plot()

all TP #1

all TP #2

all TP #3

all TN #1

all TN #2

all TN #3

all FN #1

all FN #2

all FP #1

all FP #2

XGBoost TP, others FN

XGBoost TN, others FP

SVM TP, others FN

SVM TN, others FP

Random Forest TN, others FP

Gradient Boosting TP, others FN

for row in range(16):
    print(observations_info.iloc[row]["details"])
    pp_bd_rfc = explainer_rfc.predict_parts(X_test.iloc[row,:], type = "shap")
    pp_bd_rfc.plot()

all TP #1

all TP #2

all TP #3

all TN #1

all TN #2

all TN #3

all FN #1

all FN #2

all FP #1

all FP #2

XGBoost TP, others FN

XGBoost TN, others FP

SVM TP, others FN

SVM TN, others FP

Random Forest TN, others FP

Gradient Boosting TP, others FN

for row in range(16):
    print(observations_info.iloc[row]["details"])
    pp_bd_gbc = explainer_gbc.predict_parts(X_test.iloc[row,:], type = "shap")
    pp_bd_gbc.plot()

all TP #1

all TP #2

all TP #3

all TN #1

all TN #2

all TN #3

all FN #1

all FN #2

all FP #1

all FP #2

XGBoost TP, others FN

XGBoost TN, others FP

SVM TP, others FN

SVM TN, others FP

Random Forest TN, others FP

Gradient Boosting TP, others FN